why encodeURIComponent() for data in body of POST XHR request?

[Peter Michaux]

Hi,

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

Thanks,
Peter

[David Mark]

On Aug 2, 12:43 am, Peter Michaux <[e-mail]:

Hi,

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

Thanks,
Peter

URL encoding is simply the standard for name-value pairs in an HTTP
request. GET or POST doesn't enter into it (the pairs look exactly
the same in a POST request as they do when tacked onto the query
string for a GET.)

For example, imagine what would happen if you didn't encode the pairs
and one of the values had an ampersand in it.

[Evertjan.]

David Mark wrote on 02 aug 2007 in comp.lang.javascript:

On Aug 2, 12:43 am, Peter Michaux <[e-mail]:

>>
All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

URL encoding is simply the standard for name-value pairs in an HTTP
request. GET or POST doesn't enter into it (the pairs look exactly
the same in a POST request as they do when tacked onto the query
string for a GET.)
For example, imagine what would happen if you didn't encode the pairs
and one of the values had an ampersand in it.

When using POST, nothing would, I would imagine.
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[David Mark]

On Aug 2, 3:18 am, "Evertjan." <[e-mail]:

David Mark wrote on 02 aug 2007 in comp.lang.javascript:

On Aug 2, 12:43 am, Peter Michaux <[e-mail]:

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

URL encoding is simply the standard for name-value pairs in an HTTP
request. GET or POST doesn't enter into it (the pairs look exactly
the same in a POST request as they do when tacked onto the query
string for a GET.)
For example, imagine what would happen if you didn't encode the pairs
and one of the values had an ampersand in it.

When using POST, nothing would, I would imagine.

What does that mean? You can certainly POST form data with ampersands
and they are normally encoded the same way as GET requests.

[Evertjan.]

David Mark wrote on 02 aug 2007 in comp.lang.javascript:

On Aug 2, 3:18 am, "Evertjan." <[e-mail]:

>David Mark wrote on 02 aug 2007 in comp.lang.javascript:

On Aug 2, 12:43 am, Peter Michaux <[e-mail]:

>All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding
is required in the case of a GET request and the form data is
attached as a URI query string; however, why is the encoding
necessary for POST requests?

URL encoding is simply the standard for name-value pairs in an HTTP
request. GET or POST doesn't enter into it (the pairs look exactly
the same in a POST request as they do when tacked onto the query
string for a GET.)
For example, imagine what would happen if you didn't encode the
pairs and one of the values had an ampersand in it.

When using POST, nothing would, I would imagine.

What does that mean?

Eh?
Nothing extaordinary would happen.

You can certainly POST form data with ampersands
and they are normally encoded the same way as GET requests.

With a simple querystring encoding is not automatic.

--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[Bart Van der Donck]

Peter Michaux wrote:

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

Because POST uses the same characters to split ('&' and '=') by
default. The method doesn't matter; the data is just offered in
another part of the request.

GET looks like this (header):

GET /file.html?name=Bart&nr=4 HTTP/1.0

while default POST looks like this (body):

POST /file.html HTTP/1.0
Content-length: 15

name=Bart&nr=4

It's like in email conventions, where stuff can be stored in header or
in body.

You can see this mechanism very well in Perl's CGI processing, where
GET reads the input from the URL (a line in the header of the
request), while POST takes it from <STDIN(body of request). The
further processing is identical; and the URL decoding is an important
step here.

I've seen the term "un-webify" for this kind of processing rather than
"URL-decoding"; the latter indeed sounds like it refers to GET only.

But the encoding of POST-ed data is only the default behaviour of the
browser (which is done in Ajax "by hand" in such libraries).

<form method="post">

actually means:

<form method="post"
enctype="application/x-www-form-urlencoded">

But it's possible to disable this URL-encoding for POST-ed data,
mostly to transfer (binary) files to the gateway software. In the
following example, you tell the form not to encode anything:

<form method="post" enctype="multipart/form-data">

This is the only case where 'é' will be passed as 'é'; under default
GET/POST rules it will always be passed as '%E9'.

Again, this is identical to email; all mail attachments are sent using
multipart (but, unlike HTTP, they are additionally base64-encoded).

So as a general conclusion: Ajax libraries must invoke
encodeURIComponent() when sending POST-requests in the "application/x-
www-form-urlencoded" encoding type. They must always invoke
encodeURIComponent() for GET. They must not invoke
encodeURIComponent() in the POST "multipart/form-data" type.

Obviously, it also depends on how the gateway software is configured
how to handle incoming data (this will be URL decoded by default).

Hope this helps,

--
Bart

[Bart Van der Donck]

"Evertjan." wrote:

David Mark wrote:

>You can certainly POST form data with ampersands
and they are normally encoded the same way as GET requests.

With a simple querystring encoding is not automatic.

URL-encoding is 'automatic' in browsers as long as you don't override
this setting. POST or GET doesn't matter; browser will URL-encode all
form values by default before sending them to server.

If you don't want this to happen, you should use

<form method="post" enctype="multipart/form-data">

to receive non-encoded data at server.

--
Bart

[Peter Michaux]

On Aug 1, 10:19 pm, David Mark <[e-mail]:

On Aug 2, 12:43 am, Peter Michaux <[e-mail]:

Hi,

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

Thanks,
Peter

URL encoding is simply the standard for name-value pairs in an HTTP
request. GET or POST doesn't enter into it (the pairs look exactly
the same in a POST request as they do when tacked onto the query
string for a GET.)

For example, imagine what would happen if you didn't encode the pairs
and one of the values had an ampersand in it.

Thanks, David. Now it is obvious using '&' as a separator implies the
keys and values can't have '&'. I've let myself be tripped up by
thinking these requests are more magical than just text transfer
before.

Peter

[Peter Michaux]

On Aug 2, 2:39 am, Bart Van der Donck <[e-mail]:

Peter Michaux wrote:

All Ajax libraries I've read use encodeURIComponent() on the name-
value pairs extracted from forms before POST ing the result to the
server with and xmlhttprequest. I can understand why this encoding is
required in the case of a GET request and the form data is attached as
a URI query string; however, why is the encoding necessary for POST
requests?

Because POST uses the same characters to split ('&' and '=') by
default. The method doesn't matter; the data is just offered in
another part of the request.

GET looks like this (header):

GET /file.html?name=Bart&nr=4 HTTP/1.0

while default POST looks like this (body):

POST /file.html HTTP/1.0
Content-length: 15

name=Bart&nr=4

It's like in email conventions, where stuff can be stored in header or
in body.

You can see this mechanism very well in Perl's CGI processing, where
GET reads the input from the URL (a line in the header of the
request), while POST takes it from <STDIN(body of request). The
further processing is identical; and the URL decoding is an important
step here.

I've seen the term "un-webify" for this kind of processing rather than
"URL-decoding"; the latter indeed sounds like it refers to GET only.

But the encoding of POST-ed data is only the default behaviour of the
browser (which is done in Ajax "by hand" in such libraries).

<form method="post">

actually means:

<form method="post"
enctype="application/x-www-form-urlencoded">

But it's possible to disable this URL-encoding for POST-ed data,
mostly to transfer (binary) files to the gateway software. In the
following example, you tell the form not to encode anything:

<form method="post" enctype="multipart/form-data">

This is the only case where 'é' will be passed as 'é'; under default
GET/POST rules it will always be passed as '%E9'.

Again, this is identical to email; all mail attachments are sent using
multipart (but, unlike HTTP, they are additionally base64-encoded).

So as a general conclusion: Ajax libraries must invoke
encodeURIComponent() when sending POST-requests in the "application/x-
www-form-urlencoded" encoding type. They must always invoke
encodeURIComponent() for GET. They must not invoke
encodeURIComponent() in the POST "multipart/form-data" type.

Obviously, it also depends on how the gateway software is configured
how to handle incoming data (this will be URL decoded by default).

Hope this helps,

It helps a lot. Thanks.

Peter

[Evertjan.]

Bart Van der Donck wrote on 02 aug 2007 in comp.lang.javascript:

"Evertjan." wrote:

David Mark wrote:

>>You can certainly POST form data with ampersands
and they are normally encoded the same way as GET requests.

With a simple querystring encoding is not automatic.

URL-encoding is 'automatic' in browsers as long as you don't override
this setting. POST or GET doesn't matter; browser will URL-encode all
form values by default before sending them to server.

If you don't want this to happen, you should use

<form method="post" enctype="multipart/form-data">

to receive non-encoded data at server.

I wrote:

>With a simple querystring encoding is not automatic.

var q = 'text=me&you'
location.href = 'http://cnn.com/?' + q

will be encoded?
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[Bart Van der Donck]

"Evertjan." wrote:

Bart Van der Donck wrote on 02 aug 2007 in comp.lang.javascript:

>URL-encoding is 'automatic' in browsers as long as you don't
override this setting. POST or GET doesn't matter; browser will
URL-encode all form values by default before sending them to
server.
If you don't want this to happen, you should use
<form method="post" enctype="multipart/form-data">
to receive non-encoded data at server.

var q = 'text=me&you'
location.href = 'http://cnn.com/?'+ q

will be encoded?

No, that's why Ajax libraries should use encodeURIComponent() here :)

--
Bart